Add percentage diff column to compare subcommand #1333
Conversation
b-deam
added
enhancement
|
FYI @pquentin, I liked your suggestion on statistically significant highlights, but perhaps better tackled in a seperate PR :-) I just want a quick way to see diff % at a glance for now. |
| ] | ||
| else: | ||
| return [] | ||
|
|
||
| def _diff(self, baseline, contender, treat_increase_as_improvement, formatter=lambda x: x): | ||
| def _diff(self, baseline, contender, treat_increase_as_improvement, formatter=lambda x: x, as_percentage=False): |
There was a problem hiding this comment.
For what it's worth, on my first review, I hesitated to mention that adding a boolean parameter to a function is usually considered to be a code smell. But now that Daniel suggested a refactoring, I'm less shy. :) https://adamj.eu/tech/2021/07/10/python-type-hints-how-to-avoid-the-boolean-trap/ explains why that's a problem and what the alternatives are in modern Python.
(It's fine in this case as it's a small private function only called twice: I'm not asking to change your code. But maybe you'll find those references useful.)
There was a problem hiding this comment.
Thank you for the references! They're great. It's funny because my first (uncommitted) approach was actually to implement this as a seperate private function called diff_percentage, but I was worried that there was too much code duplication.
Adds a new column showing the percentage difference between a
baselineandcontender.Closes #1328
Looks like:
Example compare
$ esrally compare --baseline b7418ff8-0e4a-4b27-b6f6-226524c313da --contender a4af31ef-076d-4c60-8d6e-3dc5aa4880a4 ____ ____ / __ \____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\__,_/_/_/\__, / /____/ Comparing baseline Race ID: b7418ff8-0e4a-4b27-b6f6-226524c313da Race timestamp: 2021-09-13 20:01:13 Challenge: append-no-conflicts Car: defaults+basic-license User tags: license=basic, name=noaa-append-defaults-1node, race-configs-id=race-configs-group-2.json, setup=bare-basic with contender Race ID: a4af31ef-076d-4c60-8d6e-3dc5aa4880a4 Race timestamp: 2021-09-14 20:00:55 Challenge: append-no-conflicts Car: defaults+basic-license User tags: license=basic, name=noaa-append-defaults-1node, race-configs-id=race-configs-group-2.json, setup=bare-basic ------------------------------------------------------ _______ __ _____ / ____(_)___ ____ _/ / / ___/_________ ________ / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \ / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/ /_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/ ------------------------------------------------------ | Metric | Task | Baseline | Contender | Diff | Unit | Diff % | |--------------------------------------------------------------:|-----------------------------------------------------:|------------:|------------:|---------:|-------:|---------:| | Cumulative indexing time of primary shards | | 42.916 | 41.8058 | -1.11023 | min | 0.00% | | Min cumulative indexing time across primary shard | | 0.0164333 | 0.0184333 | 0.002 | min | 0.00% | | Median cumulative indexing time across primary shard | | 21.458 | 20.9029 | -0.55512 | min | 0.00% | | Max cumulative indexing time across primary shard | | 42.8996 | 41.7873 | -1.11223 | min | 0.00% | | Cumulative indexing throttle time of primary shards | | 0 | 0 | 0 | min | 0.00% | | Min cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% | | Median cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% | | Max cumulative indexing throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% | | Cumulative merge time of primary shards | | 21.5249 | 25.0318 | 3.50693 | min | 0.00% | | Cumulative merge count of primary shards | | 85 | 85 | 0 | | 0.00% | | Min cumulative merge time across primary shard | | 0 | 0 | 0 | min | 0.00% | | Median cumulative merge time across primary shard | | 10.7624 | 12.5159 | 1.75347 | min | 0.00% | | Max cumulative merge time across primary shard | | 21.5249 | 25.0318 | 3.50693 | min | 0.00% | | Cumulative merge throttle time of primary shards | | 0.76065 | 0.516683 | -0.24397 | min | 0.00% | | Min cumulative merge throttle time across primary shard | | 0 | 0 | 0 | min | 0.00% | | Median cumulative merge throttle time across primary shard | | 0.380325 | 0.258342 | -0.12198 | min | 0.00% | | Max cumulative merge throttle time across primary shard | | 0.76065 | 0.516683 | -0.24397 | min | 0.00% | | Cumulative refresh time of primary shards | | 0.450417 | 0.472683 | 0.02227 | min | 0.00% | | Cumulative refresh count of primary shards | | 56 | 55 | -1 | | -1.79% | | Min cumulative refresh time across primary shard | | 0.0007 | 0.00101667 | 0.00032 | min | 0.00% | | Median cumulative refresh time across primary shard | | 0.225208 | 0.236342 | 0.01113 | min | 0.00% | | Max cumulative refresh time across primary shard | | 0.449717 | 0.471667 | 0.02195 | min | 0.00% | | Cumulative flush time of primary shards | | 1.41467 | 1.41857 | 0.0039 | min | 0.00% | | Cumulative flush count of primary shards | | 30 | 31 | 1 | | 3.33% | | Min cumulative flush time across primary shard | | 0.00433333 | 0.00403333 | -0.0003 | min | 0.00% | | Median cumulative flush time across primary shard | | 0.707333 | 0.709283 | 0.00195 | min | 0.00% | | Max cumulative flush time across primary shard | | 1.41033 | 1.41453 | 0.0042 | min | 0.00% | | Total Young Gen GC time | | 23.145 | 23.159 | 0.014 | s | 0.00% | | Total Young Gen GC count | | 2806 | 2829 | 23 | | 0.82% | | Total Old Gen GC time | | 0 | 0 | 0 | s | 0.00% | | Total Old Gen GC count | | 0 | 0 | 0 | | 0.00% | | Store size | | 5.76299 | 5.76245 | -0.00054 | GB | 0.00% | | Translog size | | 1.02445e-07 | 1.02445e-07 | 0 | GB | 0.00% | | Heap used for segments | | 0 | 0 | 0 | MB | 0.00% | | Heap used for doc values | | 0 | 0 | 0 | MB | 0.00% | | Heap used for terms | | 0 | 0 | 0 | MB | 0.00% | | Heap used for norms | | 0 | 0 | 0 | MB | 0.00% | | Heap used for points | | 0 | 0 | 0 | MB | 0.00% | | Heap used for stored fields | | 0 | 0 | 0 | MB | 0.00% | | Segment count | | 5 | 6 | 1 | | 20.00% | | Min Throughput | index | 82670.4 | 84511.3 | 1840.92 | docs/s | 2.23% | | Mean Throughput | index | 84967.3 | 87829.5 | 2862.16 | docs/s | 3.37% | | Median Throughput | index | 84751.5 | 87905.5 | 3154.04 | docs/s | 3.72% | | Max Throughput | index | 87420.6 | 89912.9 | 2492.24 | docs/s | 2.85% | | 50th percentile latency | index | 391.967 | 376.869 | -15.0971 | ms | -3.85% | | 90th percentile latency | index | 648.478 | 613.634 | -34.8438 | ms | -5.37% | | 99th percentile latency | index | 1474.85 | 1540.88 | 66.0323 | ms | 4.48% | | 99.9th percentile latency | index | 1905.24 | 2037.03 | 131.787 | ms | 6.92% | | 100th percentile latency | index | 2363.23 | 2792.3 | 429.074 | ms | 18.16% | | 50th percentile service time | index | 391.967 | 376.869 | -15.0971 | ms | -3.85% | | 90th percentile service time | index | 648.478 | 613.634 | -34.8438 | ms | -5.37% | | 99th percentile service time | index | 1474.85 | 1540.88 | 66.0323 | ms | 4.48% | | 99.9th percentile service time | index | 1905.24 | 2037.03 | 131.787 | ms | 6.92% | | 100th percentile service time | index | 2363.23 | 2792.3 | 429.074 | ms | 18.16% | | error rate | index | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_big_range | 5.99111 | 5.93585 | -0.05526 | ops/s | -0.92% | | Mean Throughput | range_field_big_range | 5.99677 | 5.97643 | -0.02033 | ops/s | -0.34% | | Median Throughput | range_field_big_range | 5.99745 | 5.9813 | -0.01615 | ops/s | -0.27% | | Max Throughput | range_field_big_range | 5.99852 | 5.98903 | -0.00949 | ops/s | -0.16% | | 50th percentile latency | range_field_big_range | 127.223 | 150.42 | 23.1971 | ms | 18.23% | | 90th percentile latency | range_field_big_range | 134.465 | 158.395 | 23.9304 | ms | 17.80% | | 99th percentile latency | range_field_big_range | 142.021 | 177.605 | 35.5844 | ms | 25.06% | | 100th percentile latency | range_field_big_range | 145.739 | 181.035 | 35.2965 | ms | 24.22% | | 50th percentile service time | range_field_big_range | 125.751 | 148.428 | 22.6769 | ms | 18.03% | | 90th percentile service time | range_field_big_range | 132.99 | 156.083 | 23.0927 | ms | 17.36% | | 99th percentile service time | range_field_big_range | 140.468 | 176.091 | 35.6229 | ms | 25.36% | | 100th percentile service time | range_field_big_range | 144.11 | 179.583 | 35.4732 | ms | 24.62% | | error rate | range_field_big_range | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_small_range | 10.0127 | 10.0113 | -0.00148 | ops/s | -0.01% | | Mean Throughput | range_field_small_range | 10.0275 | 10.0243 | -0.00318 | ops/s | -0.03% | | Median Throughput | range_field_small_range | 10.0216 | 10.0192 | -0.00241 | ops/s | -0.02% | | Max Throughput | range_field_small_range | 10.0738 | 10.0659 | -0.00787 | ops/s | -0.08% | | 50th percentile latency | range_field_small_range | 39.2806 | 41.8325 | 2.55189 | ms | 6.50% | | 90th percentile latency | range_field_small_range | 42.5939 | 44.8281 | 2.23421 | ms | 5.25% | | 99th percentile latency | range_field_small_range | 45.2979 | 47.7983 | 2.50035 | ms | 5.52% | | 100th percentile latency | range_field_small_range | 46.5945 | 49.5504 | 2.95588 | ms | 6.34% | | 50th percentile service time | range_field_small_range | 37.8183 | 40.334 | 2.51573 | ms | 6.65% | | 90th percentile service time | range_field_small_range | 40.748 | 43.0768 | 2.32881 | ms | 5.72% | | 99th percentile service time | range_field_small_range | 43.7154 | 45.9446 | 2.22915 | ms | 5.10% | | 100th percentile service time | range_field_small_range | 45.0396 | 47.6532 | 2.61357 | ms | 5.80% | | error rate | range_field_small_range | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_conjunction_big_range_small_term_query | 9.98741 | 10.0072 | 0.01975 | ops/s | 0.20% | | Mean Throughput | range_field_conjunction_big_range_small_term_query | 9.99499 | 10.0154 | 0.02044 | ops/s | 0.20% | | Median Throughput | range_field_conjunction_big_range_small_term_query | 9.99601 | 10.0122 | 0.01622 | ops/s | 0.16% | | Max Throughput | range_field_conjunction_big_range_small_term_query | 9.99762 | 10.0416 | 0.04397 | ops/s | 0.44% | | 50th percentile latency | range_field_conjunction_big_range_small_term_query | 44.9031 | 47.7657 | 2.86259 | ms | 6.38% | | 90th percentile latency | range_field_conjunction_big_range_small_term_query | 56.1899 | 56.9081 | 0.71816 | ms | 1.28% | | 99th percentile latency | range_field_conjunction_big_range_small_term_query | 57.8198 | 58.4238 | 0.60402 | ms | 1.04% | | 100th percentile latency | range_field_conjunction_big_range_small_term_query | 58.9602 | 61.4708 | 2.51059 | ms | 4.26% | | 50th percentile service time | range_field_conjunction_big_range_small_term_query | 43.4839 | 46.3866 | 2.90265 | ms | 6.68% | | 90th percentile service time | range_field_conjunction_big_range_small_term_query | 54.5794 | 55.3553 | 0.77593 | ms | 1.42% | | 99th percentile service time | range_field_conjunction_big_range_small_term_query | 55.9615 | 56.8048 | 0.84333 | ms | 1.51% | | 100th percentile service time | range_field_conjunction_big_range_small_term_query | 56.3711 | 60.1427 | 3.77162 | ms | 6.69% | | error rate | range_field_conjunction_big_range_small_term_query | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_conjunction_small_range_small_term_query | 10.0113 | 10.0107 | -0.00058 | ops/s | -0.01% | | Mean Throughput | range_field_conjunction_small_range_small_term_query | 10.0246 | 10.0231 | -0.0015 | ops/s | -0.01% | | Median Throughput | range_field_conjunction_small_range_small_term_query | 10.0196 | 10.0184 | -0.00115 | ops/s | -0.01% | | Max Throughput | range_field_conjunction_small_range_small_term_query | 10.0675 | 10.0636 | -0.00391 | ops/s | -0.04% | | 50th percentile latency | range_field_conjunction_small_range_small_term_query | 44.5273 | 40.44 | -4.08727 | ms | -9.18% | | 90th percentile latency | range_field_conjunction_small_range_small_term_query | 48.2358 | 50.509 | 2.27326 | ms | 4.71% | | 99th percentile latency | range_field_conjunction_small_range_small_term_query | 51.3412 | 53.9716 | 2.63046 | ms | 5.12% | | 100th percentile latency | range_field_conjunction_small_range_small_term_query | 53.4269 | 55.1778 | 1.75082 | ms | 3.28% | | 50th percentile service time | range_field_conjunction_small_range_small_term_query | 42.8639 | 38.4964 | -4.3675 | ms | -10.19% | | 90th percentile service time | range_field_conjunction_small_range_small_term_query | 46.4451 | 48.7918 | 2.3467 | ms | 5.05% | | 99th percentile service time | range_field_conjunction_small_range_small_term_query | 49.7795 | 51.7462 | 1.9667 | ms | 3.95% | | 100th percentile service time | range_field_conjunction_small_range_small_term_query | 51.8831 | 53.5194 | 1.63634 | ms | 3.15% | | error rate | range_field_conjunction_small_range_small_term_query | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_conjunction_small_range_big_term_query | 4.00177 | 4.00249 | 0.00072 | ops/s | 0.02% | | Mean Throughput | range_field_conjunction_small_range_big_term_query | 4.00382 | 4.00538 | 0.00156 | ops/s | 0.04% | | Median Throughput | range_field_conjunction_small_range_big_term_query | 4.00305 | 4.0043 | 0.00125 | ops/s | 0.03% | | Max Throughput | range_field_conjunction_small_range_big_term_query | 4.01058 | 4.0148 | 0.00422 | ops/s | 0.11% | | 50th percentile latency | range_field_conjunction_small_range_big_term_query | 135.953 | 139.048 | 3.09467 | ms | 2.28% | | 90th percentile latency | range_field_conjunction_small_range_big_term_query | 155.652 | 157.501 | 1.84982 | ms | 1.19% | | 99th percentile latency | range_field_conjunction_small_range_big_term_query | 158.46 | 159.868 | 1.40749 | ms | 0.89% | | 100th percentile latency | range_field_conjunction_small_range_big_term_query | 161.26 | 164.711 | 3.45166 | ms | 2.14% | | 50th percentile service time | range_field_conjunction_small_range_big_term_query | 134.187 | 137.308 | 3.12155 | ms | 2.33% | | 90th percentile service time | range_field_conjunction_small_range_big_term_query | 154.091 | 156.032 | 1.94082 | ms | 1.26% | | 99th percentile service time | range_field_conjunction_small_range_big_term_query | 156.599 | 158.448 | 1.84818 | ms | 1.18% | | 100th percentile service time | range_field_conjunction_small_range_big_term_query | 159.702 | 162.979 | 3.27708 | ms | 2.05% | | error rate | range_field_conjunction_small_range_big_term_query | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_conjunction_big_range_big_term_query | 1.00065 | 1.00056 | -8e-05 | ops/s | -0.01% | | Mean Throughput | range_field_conjunction_big_range_big_term_query | 1.00138 | 1.00121 | -0.00017 | ops/s | -0.02% | | Median Throughput | range_field_conjunction_big_range_big_term_query | 1.0011 | 1.00096 | -0.00014 | ops/s | -0.01% | | Max Throughput | range_field_conjunction_big_range_big_term_query | 1.00384 | 1.00336 | -0.00048 | ops/s | -0.05% | | 50th percentile latency | range_field_conjunction_big_range_big_term_query | 618.553 | 681.964 | 63.4114 | ms | 10.25% | | 90th percentile latency | range_field_conjunction_big_range_big_term_query | 638.972 | 692.708 | 53.7362 | ms | 8.41% | | 99th percentile latency | range_field_conjunction_big_range_big_term_query | 656.6 | 707.452 | 50.8522 | ms | 7.74% | | 100th percentile latency | range_field_conjunction_big_range_big_term_query | 691.056 | 718.323 | 27.2667 | ms | 3.95% | | 50th percentile service time | range_field_conjunction_big_range_big_term_query | 616.538 | 680.137 | 63.5992 | ms | 10.32% | | 90th percentile service time | range_field_conjunction_big_range_big_term_query | 636.973 | 690.979 | 54.0058 | ms | 8.48% | | 99th percentile service time | range_field_conjunction_big_range_big_term_query | 655.027 | 706.019 | 50.9919 | ms | 7.78% | | 100th percentile service time | range_field_conjunction_big_range_big_term_query | 689.254 | 716.412 | 27.1574 | ms | 3.94% | | error rate | range_field_conjunction_big_range_big_term_query | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_disjunction_small_range_small_term_query | 10.003 | 10.0025 | -0.00048 | ops/s | 0.00% | | Mean Throughput | range_field_disjunction_small_range_small_term_query | 10.0064 | 10.0054 | -0.00105 | ops/s | -0.01% | | Median Throughput | range_field_disjunction_small_range_small_term_query | 10.0052 | 10.0042 | -0.00096 | ops/s | -0.01% | | Max Throughput | range_field_disjunction_small_range_small_term_query | 10.0179 | 10.0147 | -0.00317 | ops/s | -0.03% | | 50th percentile latency | range_field_disjunction_small_range_small_term_query | 50.2813 | 49.4949 | -0.78633 | ms | -1.56% | | 90th percentile latency | range_field_disjunction_small_range_small_term_query | 61.1144 | 55.6472 | -5.46725 | ms | -8.95% | | 99th percentile latency | range_field_disjunction_small_range_small_term_query | 66.2176 | 62.2997 | -3.91793 | ms | -5.92% | | 100th percentile latency | range_field_disjunction_small_range_small_term_query | 67.3406 | 66.308 | -1.03259 | ms | -1.53% | | 50th percentile service time | range_field_disjunction_small_range_small_term_query | 48.809 | 47.9481 | -0.86087 | ms | -1.76% | | 90th percentile service time | range_field_disjunction_small_range_small_term_query | 59.6564 | 53.9495 | -5.70696 | ms | -9.57% | | 99th percentile service time | range_field_disjunction_small_range_small_term_query | 64.4142 | 60.4733 | -3.9409 | ms | -6.12% | | 100th percentile service time | range_field_disjunction_small_range_small_term_query | 65.7147 | 64.8426 | -0.87214 | ms | -1.33% | | error rate | range_field_disjunction_small_range_small_term_query | 0 | 0 | 0 | % | 0.00% | | Min Throughput | range_field_disjunction_big_range_small_term_query | 6.00075 | 5.93597 | -0.06477 | ops/s | -1.08% | | Mean Throughput | range_field_disjunction_big_range_small_term_query | 6.00157 | 5.95256 | -0.04902 | ops/s | -0.82% | | Median Throughput | range_field_disjunction_big_range_small_term_query | 6.00123 | 5.94617 | -0.05506 | ops/s | -0.92% | | Max Throughput | range_field_disjunction_big_range_small_term_query | 6.00428 | 5.99125 | -0.01304 | ops/s | -0.22% | | 50th percentile latency | range_field_disjunction_big_range_small_term_query | 141.781 | 781.116 | 639.335 | ms | 450.93% | | 90th percentile latency | range_field_disjunction_big_range_small_term_query | 149.351 | 957.46 | 808.109 | ms | 541.08% | | 99th percentile latency | range_field_disjunction_big_range_small_term_query | 156.692 | 1142.2 | 985.506 | ms | 628.94% | | 100th percentile latency | range_field_disjunction_big_range_small_term_query | 166.914 | 1148.27 | 981.353 | ms | 587.94% | | 50th percentile service time | range_field_disjunction_big_range_small_term_query | 140.144 | 165.764 | 25.6204 | ms | 18.28% | | 90th percentile service time | range_field_disjunction_big_range_small_term_query | 147.439 | 173.604 | 26.1641 | ms | 17.75% | | 99th percentile service time | range_field_disjunction_big_range_small_term_query | 153.918 | 194.131 | 40.2135 | ms | 26.13% | | 100th percentile service time | range_field_disjunction_big_range_small_term_query | 165.326 | 195.512 | 30.1859 | ms | 18.26% | | error rate | range_field_disjunction_big_range_small_term_query | 0 | 0 | 0 | % | 0.00% | ------------------------------- [INFO] SUCCESS (took 3 seconds) -------------------------------